-
Notifications
You must be signed in to change notification settings - Fork 6.8k
Conversation
Change-Id: I4da04bf4502425134a466823fb5f73da2d7a419b
Are you trying to use this PR for testing the issue? If that's the case it doesn't make sense coz its a costly experiment in that case. |
Some flaky tests always fail in CI but cannot reproduce easily from local even with docker env on EC2. So debug with the CI environment can be more efficiency to find the root cause. |
@xinyu-intel this is not a proper use of public resources. If you have issues with reproducing your issues on local then you should ask on dev@ channel thats a good place to start. Using CI for debugging causes a fresh run of all 11 test suits on your PR each time you push new changes. A single run of all 11 tests costs approximately 200$. Please be mindful of that. |
@xinyu-intel you can use the ci/build.py tool to easily reproduce what runs on CI. it's dockerized so it's going to produce the same result with very high probability: https://cwiki.apache.org/confluence/display/MXNET/Reproducing+test+results#Reproducingtestresults-2.1.Build |
okay, I'll ask in dev list next time. |
@larroy I have already used this script and it is very easy for use. However, in contrast with "very high probability", this case also cannot be reproduced with |
Let's skip this test first :) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
@ChaiBapchya @access2rohit @larroy Really appreciate your suggestions and frugal spirit for MXNet development. I think this PR is not the first time for contributors to use CI for debugging purpose, and most likely it will not be the last one. I just took two simple filters on the PR list: I believe @xinyu-intel has tried his best to reproduce this issue on both local machine and also on EC2 instances to better emulate the CI environment. But unfortunately, he cannot reproduce the error and I think it's not the first time that we cannot reproduce an error out of CI. Another choice is maybe someone can contribute to the CI system so developers can only trigger just one test suits rather than all of the 11 tests suits. It will be very helpful especially to document contributors. |
@TaoLv I was already pointed out by @szha about the consequences of opening such PRs already and I closed those PRs. Also, if I had done that before doesn't mean I cannot suggest other developers not to do the same mistake as I did. Yes, if such a behaviour becomes widespread I will start a thread on @dev list. |
@access2rohit Good to learn that. |
Hi @access2rohit and @larroy, I have several suggestions about MXNet CI.
It is better to provide some machines to contributors to debug : )
Thanks. |
Option 1 is certainly not an option. The CI is a productive system and granting ssh access is a security concern. Amazon security was very explicit when we went through the security audit. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thank you for the fix : )
* Fix quantized concat when inputs are mixed int8 and uint8 Change-Id: I4da04bf4502425134a466823fb5f73da2d7a419b * skip flaky test * trigger ci [Numpy] add op full_like, c++ impl, fix zeros_like, ones_like type inference (apache#16804)
print('skipped testing quantized_conv for mkldnn cpu int8 since it is not supported yet') | ||
elif is_test_for_mkldnn(): | ||
# (TODO)Xinyu: https://github.com/apache/incubator-mxnet/issues/16830 | ||
print('skipped testing quantized_conv for mkldnn cpu since it is a flaky case') |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@xinyu-intel Any chance we switch this back on?
* Fix quantized concat when inputs are mixed int8 and uint8 Change-Id: I4da04bf4502425134a466823fb5f73da2d7a419b * skip flaky test * trigger ci
…8560) * fix centos 7 url to unblock centos-cpu & gpu pipeline * [v1.7.x] update jetson dockerfile to support CUDA 10.0 (#18339) * update dockerfile for jetson * add toolchain files * update build_jetson function * update ubuntu_julia.sh * update FindCUDAToolkit.cmake * Update centos7_python.sh * revert changes on ubuntu_julia.sh * disable TVM for gpu build * Disable TVM_OP on GPU builds Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl> * skip quantized conv flaky case (#16866) * Fix quantized concat when inputs are mixed int8 and uint8 Change-Id: I4da04bf4502425134a466823fb5f73da2d7a419b * skip flaky test * trigger ci Co-authored-by: waytrue17 <52505574+waytrue17@users.noreply.github.com> Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl> Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com>
* Fix quantized concat when inputs are mixed int8 and uint8 Change-Id: I4da04bf4502425134a466823fb5f73da2d7a419b * skip flaky test * trigger ci
* add the missing build_ubuntu_gpu_cuda101_cudnn7_mkldnn_cpp_test in runtime_functions.sh * Revert "add the missing build_ubuntu_gpu_cuda101_cudnn7_mkldnn_cpp_test in runtime_functions.sh" This reverts commit de173b0. * Revert "[CI][1.6.x] fix centos 7 url to unblock centos-cpu & gpu pipeline (#18560)" This reverts commit d271348. * fix centos 7 url to unblock centos-cpu & gpu pipeline * skip quantized conv flaky case (#16866) * Fix quantized concat when inputs are mixed int8 and uint8 Change-Id: I4da04bf4502425134a466823fb5f73da2d7a419b * skip flaky test * trigger ci * Trigger empty commit * [v1.7.x] update jetson dockerfile to support CUDA 10.0 (#18339) * update dockerfile for jetson * add toolchain files * update build_jetson function * update ubuntu_julia.sh * update FindCUDAToolkit.cmake * Update centos7_python.sh * revert changes on ubuntu_julia.sh * disable TVM for gpu build * Disable TVM_OP on GPU builds Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl> * add setuptools to ci/docker/install/requirements * add missing build_ubuntu_gpu_cuda101_cudnn7_mkldnn_cpp_test * add setuptool to docker & cpp-test build syntax error * remove erroneously added cpp tests in 1.6.x * py3 to p2 Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com> Co-authored-by: waytrue17 <52505574+waytrue17@users.noreply.github.com> Co-authored-by: Wei Chu <weichu@amazon.com> Co-authored-by: Leonard Lausen <leonard@lausen.nl>
Description
temporary skip this case.
@pengzhao-intel
#16830
Checklist
Essentials
Please feel free to remove inapplicable items for your PR.
Changes
Comments